Machine-Learning Model for Ranking Diverse Content

ABSTRACT

In one embodiment, a method includes a computing system accessing a content item associated with a content provider, the content item having a first set of attributes and a second set of attributes. The system may generate, using a first machine-learning model, a first ranking score of the content item for a user based on the first set of attributes. The system may generate cluster representations of the second set of attributes of the content item. The system may generate, using a second machine-learning model, a second ranking score of the content item for the user based on the cluster representations. The system may generate, using a third machine-learning model, a third ranking score of the content item for the user based on the first ranking score and the second ranking score. The system may select the content item for presentation to the user based on the third ranking score.

TECHNICAL FIELD

This disclosure generally relates to machine learning for ranking.

BACKGROUND

A social-networking system, which may include a social-networkingwebsite, may enable its users (such as persons or organizations) tointeract with it and with each other through it. Increasingly,social-networking systems, as well as other types of Internet-basedplatforms (e.g., web sites, e-mail systems, servers of mobile-deviceapplications, etc.), are being leveraged as content distributors.Different content providers may request the Internet-based platform todistribute the content providers' respective content to users of theplatform. Certain content providers may provide multiple content itemsto the platform and request the platform to select the ones that wouldlikely be of interest to the receiving users. The platform may use aranking model to rank the candidate content items and select the onesthat are most likely to be of interest to the user. Since differentcontent providers may request the platform to distribute different typesof content items under different circumstances, the platform may customtailor different ranking models for different needs of the differentcontent providers. However, developing custom ranking models in thismanner suffers from scalability constraints, especially for rankingmodels that require sufficiently large data samples to develop.

SUMMARY OF PARTICULAR EMBODIMENTS

Particular embodiments described herein relate to a robust rankingsystem that allows diverse, unstructured content from different contentproviders to be ranked. The machine-learning models used in the rankingsystem may not necessarily be trained on training data that arehomogeneous with respect to the input data at inference time. Inparticular embodiments, the ranking system may achieve this robustnessby conceptually separate the data attributes of a content item providedby a content provider into (1) known attributes that can be mapped to aset of common attribute types that are recognized by a first rankingmodel and (2) custom attributes that cannot be mapped or are otherwisenot recognized by that first ranking model. In particular embodiments,the known attributes may be consumed by the first ranking model togenerate a ranking score for the content item. In particularembodiments, the custom attributes may be grouped based on a clusteringmodel so that similar attributes from different content providers may betreated as being equivalent. A second ranking model may be trained toconsume the clustered custom attribute data. Since the second rankingmodel is trained to recognize the generalized cluster information, theranking system would not need custom ranking models for differentcontent providers or types of content items. In particular embodiments,a third ranking model may then take as input the ranking results fromthe first ranking model and the second ranking model to generate aranking score for the content item.

The embodiments disclosed herein are only examples, and the scope ofthis disclosure is not limited to them. Particular embodiments mayinclude all, some, or none of the components, elements, features,functions, operations, or steps of the embodiments disclosed above.Embodiments according to the invention are in particular disclosed inthe attached claims directed to a method, a storage medium, a system anda computer program product, wherein any feature mentioned in one claimcategory, e.g. method, can be claimed in another claim category, e.g.system, as well. The dependencies or references back in the attachedclaims are chosen for formal reasons only. However, any subject matterresulting from a deliberate reference back to any previous claims (inparticular multiple dependencies) can be claimed as well, so that anycombination of claims and the features thereof are disclosed and can beclaimed regardless of the dependencies chosen in the attached claims.The subject-matter which can be claimed comprises not only thecombinations of features as set out in the attached claims but also anyother combination of features in the claims, wherein each featurementioned in the claims can be combined with any other feature orcombination of other features in the claims. Furthermore, any of theembodiments and features described or depicted herein can be claimed ina separate claim and/or in any combination with any embodiment orfeature described or depicted herein or with any of the features of theattached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of heterogeneous content providers andcontent items.

FIG. 2 illustrates a ranking system architecture in accordance withparticular embodiments.

FIG. 3 illustrates an example method for ranking a content item usingthe ranking system.

FIG. 4 illustrates an example network environment associated with asocial-networking system.

FIG. 5 illustrates an example social graph.

FIG. 6 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Particular embodiments described herein relate to a robust rankingsystem that allows diverse, unstructured content from different contentproviders to be ranked. The machine-learning models used in the rankingsystem may not necessarily be trained on training data that arehomogeneous with respect to the input data at inference time. Inparticular embodiments, the ranking system may achieve this robustnessby conceptually separate the data attributes of a content item providedby a content provider into (1) known attributes that can be mapped to aset of common attribute types that are recognized by a first rankingmodel and (2) custom attributes that cannot be mapped or are otherwisenot recognized by that first ranking model. In particular embodiments,the known attributes may be consumed by the first ranking model togenerate a ranking score for the content item. In particularembodiments, the custom attributes may be grouped based on a clusteringmodel so that similar attributes from different content providers may betreated as being equivalent. A second ranking model may be trained toconsume the clustered custom attribute data. Since the second rankingmodel is trained to recognize the generalized cluster information, theranking system would not need custom ranking models for differentcontent providers or types of content items. In particular embodiments,a third ranking model may then take as input the ranking results fromthe first ranking model and the second ranking model to generate aranking score for the content item.

A ranking system may need to be sufficiently robust to handle contentfrom a variety of sources. FIG. 1 illustrates an example ofheterogeneous content providers and content items. Three different typesof content providers are illustrated: an electronics store 110, anonline music distributor 120, and a news provider 130. The differenttypes of content providers may request the ranking system to rankdifferent types of content. For example, the electronic store 110 maywish to rank its products, the online music distributor 120 may wish torank its albums, and the news provider may wish to rank its newsarticles. Other types of content providers may also ask the system torank any other type of content, including photos, audio, video, streamedcontent, personal profiles, comments, goods and services, or any othertypes of content. In addition, some content providers may have esotericcontent types that are highly specific to them (e.g., choosing whichversion of their homepage to show each user based on ranking score).Each content type may have unique attributes. For example, theelectronics store 110 may have a product content type 113 that includesthe following attribute types for describing each product: item name,brand, and specification. The online music distributor 120 may have analbum content type 123 that includes the following attribute types fordescribing each album: album name, singer, genre, and release date. Thenews provider 130 may have a news content type 133 that includes thefollowing attribute types for describing each news item: title, subject,category, and subcategory. From the examples shown, it should beapparent that the attributes of different content types, especiallythose of different content providers, may be very different. Whiledifferences exist, one should also note that some attributes may beconceptually similar. For example, the item name of the products contenttype 113, the album name of the album content type 123, and the title ofthe news content type 133 may all be considered as labels for theirrespective content types.

In addition to differences between content types, different contentproviders may have different content-ranking objectives and may wish theranking system to consider other types of context information. Anexample of different content-ranking objectives or outcome includeoptimizing for user clicks (e.g., a user clicking on an article to seethe full article on the content provider's website), usercomments/engagement (e.g., a user commenting on, sharing, or “liking”the content item presented through a social-networking platform), userinstalls (e.g., a user agreeing to install a software applicationassociated with the presented content item), user downloads (e.g., auser downloading or streaming a song or video), user re-blogs (e.g., auser re-posting or re-distributing the content item), user purchases,etc. Different content providers may also wish to target different typesof demographics. For example, the online music distributor 120 may wishto target teens for its pop songs, and the news provider 130 may wish totarget industry professionals for its financial news. Additionally,different content providers may wish the ranking system to considerdifferent types of context information that may influence a particularranking. This may include, for example, recent or current activities ofthe user for whom the content item is being targeted. For instance, theelectronics store 110 may want the ranking system to consider the itemsthat are currently in the target user's cart, the online musicdistributor 120 may want the ranking system to consider the typicalsongs that the target user listens to, and the news provider 130 maywant the ranking system to consider the types of news articles that thetarget user read yesterday.

To accommodate the needs of different content providers, particularembodiments may build custom rankers for each content provider. This isillustrated in FIG. 1 by Ranker A 115, Ranker B 125, and Ranker C 135being used for ranking the product content type 113, album content type123, and the news content type 133, respectively. Each of Ranker A 115,Ranker B 125, and Ranker C 135 may process their respective input datato output rank 119, rank 129, and rank 139, respectively.

However, given the potentially endless differences in content typesand/or other information that content providers may ask the rankingsystem to process, building a custom ranking model for each contentprovider and/or each content type may be costly, time-consuming, and notscalable. Further, a new content provider may not have sufficiently richdata to adequately train its custom model, at least in the beginning,and therefore the effectiveness and efficiency of the model may not beoptimal until sufficient data has been gathered (this may be referred toas the “cold-start problem”). An alternative may be to build a singleranking model using the available data from different content providers,but doing so may require certain generalizations to be made (e.g., notconsidering certain more esoteric, but potentially highlypredictive/relevant, attributes). The single model may also beover-fitted to common content types and under-fitted for relatively morerare content types. Furthermore, since the single model is trained onexisting data, it may not be sufficiently calibrated to process a newcontent type from a new content provider.

To address the aforementioned issues, particular embodiments provide aranking system that allows diverse, unstructured content from differentcontent providers to be ranked using machine-learning models. Inparticular embodiments, the machine-learning models may be trained onany available data associated with any content providers, even if thecontent providers and/or their respective content types differ. Oncetrained, the machine-learning models may be used to process and rank anycontent type, even if that content type is different from any of thecontent types used in training. Stated differently, the machine-learningmodels may be trained on existing data that may or may not reflect theinput data that is to be ranked at inference time. For example, themachine-learning models may be trained using existing training data,such as content items of an electronics store, but the trainingmachine-learning models may be used to rank different types of contentitems, such as those of a music distributor. This deviates fromconventional configurations where a machine-learning model is trained ondata with the same attributes types as the data that the model istrained to process at inference time. Thus, one benefit of the rankingsystem is that once it has been trained, it may be used to rank even newcontent types and provide reasonable ranking quality from the start. Soeven if the new content type lacks sufficient training data to train acustom ranking model, the new content type may be ranked using theranking system that was trained on other, existing content types.

FIG. 2 illustrates a robust ranking system 200 in accordance withparticular embodiments. The ranking system 200 may be configured to rankcontent for particular users or content consumers. Particularly insituations where there is a large volume of content (e.g., a catalogueof items, songs, or news articles), an objective of the ranking systemmay be to rank and prioritize content for specific viewers. By providingtargeted content that is relevant or of interest to its viewer, theranking system helps content providers reach their audience in anefficient and effective manner. Additionally, the platform helps focuscontent for viewers, thereby preventing information overload or fatigue.

In particular embodiments, a content provider may request the rankingsystem 200 to rank and distribute its content. The content provider mayprovide the ranking system 200 with a variety of information 210 thatmay be used for ranking content items 211. As an example, a contentprovider (e.g., music distributor) may request the ranking system 200 torank one or more content items 211 (e.g., songs) for individual users(e.g., to provide each user with personalized song recommendations). Inparticular embodiments, each content item 211 provided by the contentprovider may have a variety of attributes. Some of the attributes may beof known attribute types 212 and others may be of custom attribute types213, which may be different from the known attribute types 212. Theknown attribute types 212 may be attribute types that were used in thetraining of the machine-learning model 230. For example, if themachine-learning model 230 was trained on data that have the attribute,Item Description, the content item's 212 Item Description attribute maybe considered as a known attribute. In particular embodiments, theranking system 200 may be configured to recognize that certainattributes of the content item 211 are equivalent to a known attributethat is recognized by the machine-learning model 230. For example, theranking system 200 may recognize that the Item Name attribute type ofthe content item 211 can be mapped to the Item Label attribute type thatwas used in the training of the machine-learning model 230. Inparticular embodiments, the attributes of the content item 211 may beprovided in a structured format (e.g., XML, JSON, etc.) with informationthat may be used to transform or map the structured attributes into aformat recognized by the machine-learning model 230.

In particular embodiments, the content item 211 may also includeattributes that may be referred to as custom attribute types 213. Customattribute types 213, as used herein, refers to attribute types that werenot used in the training of the machine-learning model 230. In otherwords, custom attribute types 213 are different from the attribute typesof the training data used for training the machine-learning model 230.As previously described, content items from different providers may haveunique, custom attributes. For instance, if the content item 211 is asong, it may have custom attributes that indicate the song's reviewscore on the content provider's platform (e.g., 4 stars out of a maximumof five stars) and the song's duration. These types of attributes may beunique to the particular content provider or content type. For instance,an electronic product content item or a news content item may not have asong duration attribute, since such an attribute may not make sense inthe context of the content item. Since custom attribute types 213 maynot be recognized by the machine-learning model 230, the ranking system200 may process them differently, as will be described in further detailbelow.

In particular embodiments, the ranking system 200 may be asked to rankcontent items for a particular user, as represented by a user ID 214. Inparticular embodiments, the content provider may generally request thatits content items be distributed to users in a personalized fashion(e.g., selectively surface content items that are likely to be ofinterest to each individual user). Based on such a request, thesocial-networking platform (or other types of content-distributionplatform) may identify the particular users to whom to surface thecontent items. For example, if the content provider specified aparticular user demographic of interest, the social-networking platformmay select users accordingly. As another example, when a user requestscontent (e.g., a newsfeed) from the social-networking platform, theplatform may identify the requesting user as a potential candidate forreceiving the content items of the content provider. In both of theseexamples, the content provider does not specify the specific user towhom to distribute content items. Rather, it is the social-networking orcontent-distribution platform that is identifying the target candidateuser. In particular embodiments, the content provider may specify thespecific user to whom to distribute content items. For example, aparticular user may be browsing songs on the content provider's system,and the content provider may wish to leverage the vast data and rankingsystem 200 of the social-networking platform to predict which songs arelikely to be of interest to that user. In particular embodiments, thecontent provider may send a request (e.g., via an API call) to thesocial-networking platform, asking content items (e.g., songs) to beranked for that particular user. In this scenario, the content providermay specify the specific user to whom the content items are directed.

In particular embodiments, the ranking system 200 may further take intoconsideration context information surrounding a particular rankingrequest or the context in which content item is to be displayed. Forexample, in embodiments where the ranking of a content item ispersonalized for a particular user, the content provider may alsoprovide user context information 215 pertaining to the current contextin which the content item is to be displayed. For example, the contentprovider may provide information such as the types of content that theviewer is currently consuming or has recently consumed (e.g., types ofsongs the user recently listened to, news articles that the user iscurrently reading, software applications that has been downloaded,etc.), the items that are currently in the user's cart, search termsused by the user within a predetermined time window, and any other typeof user information or user activity data that may be relevant forpredicting the user's interest in the content items. Thesocial-networking platform or content-distribution platform that isbeing asked to perform the ranking may also retrieve, from theplatform's knowledgeable, user information or activity data that may beused to assess the current state of mind or interest of the user. Forexample, a social-networking platform may identify posts, newsfeeds, orvideos that the user recently viewed or engaged with (e.g., bycommenting, “liking,” sharing, etc.). Such user context information maybe used by the ranking system 200 to predict which of the content items211 would likely be of interest to the user, as will be described infurther detail below.

To improve ranking effectiveness, the content provider may, inparticular embodiments, provide certain metadata 216 about itself aswell, such as its company size, industry, geographic location, and anyother suitable information that may help the content provider reach itsintended audience. Furthermore, as previously described, differentcontent providers may have different ranking objectives to achievecertain desired outcomes. For example, some content providers may wishto optimize the ranking results for clicks, while others may wish tooptimize ranking for social-networking engagements (e.g., comments,shares, etc.), downloads, etc. Content providers may also know thedesired or preferred user demographics. Thus, in particular embodiments,the ranking system may also take into consideration the contentprovider's metadata 316 that indicate the content provider's preferredranking objectives, outcomes, demographics, etc.

In particular embodiments, the aforementioned data (e.g., knownattributes and custom attributes of the content items, user data, andcontent provider data) may be processed by different machine-learningmodels. With respect to the known attributes of the content items, theranking system 200 may use a machine-learning model 230 to rank thecontent items based their respective attributes that are known by themachine-learning model 230. In particular embodiments, themachine-learning model 230 may be trained using a training dataset thathas a set of known attribute types. In particular embodiments, themachine-learning model 230 may be an existing machine-learning model ofthe social-networking platform, trained to rank content items based on aset of rich, existing training data with predetermined attributes. Thetraining data may include content items from a variety of contentproviders, and the training content items may or may not reflect thetype of content items that the current content provider is asking thesystem 200 to rank. For example, the existing machine-learning model mayhave been trained using data from a wholesaler or electronics store,while the current content provider may be a music or news distributor.However, since the current content item may have attributes in commonwith those used for training the machine-learning model 230, the model230 may nevertheless be used to rank the content items based on suchattributes (albeit based on incomplete data, since the content items mayhave custom attribute types 213). In particular embodiments, eachtraining sample in the training data set may include (1) a trainingcontent item with attributes of the known types, (2) user dataassociated with a user to whom the training content item was presented,and (3) a ranking metric (e.g., whether the user clicked or downloadedthe content item) that represents how the training content item shouldbe ranked (e.g., this may be considered as the ground truth or label ofthe training sample). In particular embodiments, the machine-learningmodel 230 may be a neural network, but any other suitable types ofmachine-learning models may also be used.

In particular embodiments, the ranking system 200 may use the trainedmachine-learning model 230 to generate a preliminary rank for a givencontent item 211 based on its known attributes 212. In particularembodiments, the content provider may provide structured data that maybe used to determine how the known attributes 212 should be interpreted.For example, the content item may be a JSON blob with structuredattributes (e.g., name, description, price, timestamp, etc.) that may bemapped to attributes known or recognized by the machine-learning model230. Thus, in particular embodiments, the ranking system 200 maytransform 220 the structured attributes of the known attribute types 212into a format or data structure expected by the machine-learning model230 (which may be trained to handle null attributes). In particularembodiments, the transformed set of attributes of the known attributetype 212 may be input into the machine-learning model 230. In particularembodiments, the machine-learning model 230 may also take as input userdata associated with the user for whom the content item is being ranked.In particular embodiments, the user data may be retrieved 260 from thesocial-networking/content-distribution platform and/or the contentprovider. Based on the input data, the machine-learning model 230 maygenerate a ranking score for the corresponding content item 211 (e.g.,the ranking score may represent a relative relevance of the content itemto the particular user). The ranking score from the machine-learningmodel 230 may then be input into a final ranking model 270, which willbe described below.

In particular embodiments, the ranking system 200 may use a differentmachine-learning model 250 to process the custom attribute types 213 ofthe content item 211. As described elsewhere herein, the content itemsprovided by the current content provider may have data attributes thatdo not neatly map to any of the predetermined features recognized by themachine-learning model 230 configured for known attributes. Yet, thesecustom data attributes may nevertheless be relevant to how the contentitem should be ranked to optimize some desired metric. One way for theranking system 200 to take into consideration the custom attribute types213 is to train a machine-learning model based on the custom attributesdirectly. However, this approach may result in the system 200 needingdifferent machine-learning models for different types of content items.Thus, in particular embodiments, the ranking system 200 may insteadgeneralize the custom attribute types 213 and train a machine-learningmodel 250 based on the generalized data.

In particular embodiments, the ranking system may use a clustering model240 to cluster the custom attributes of each content item in thetraining data based on their respective similarities to other attributesacross the whole ecosystem (e.g., across the social-networking platformor content-distribution platform). In particular embodiments, similarattributes may be clustered closer together in the clustering space of ndimensions. In particular embodiments, for each custom attribute, theclustering model 240 may generate a corresponding cluster ID or vectorrepresentation of that attribute in the cluster space. In particularembodiments, the cluster representations output by the clustering model240, along with the attributes of the custom type 213, may be used totrain a machine-learning model 250 to rank content items based on theirrespective cluster representations and the associated custom dataattributes. In addition, the machine-learning model 250 may also take asinput user data associated with the user for whom the content item waspresented. In particular embodiments, each training sample in thetraining data set may include (1) a training content item (which maycorrespond to the training content items used for training the firstmachine-learning model 230) with attributes of the custom types, (2)user data associated with a user to whom the training content item waspresented, and (3) a ranking metric (e.g., whether the user clicked ordownloaded the content item) that represents how the training contentitem should be ranked (e.g., this may be considered as the ground truthor label of the raining sample). In particular embodiments, themachine-learning model 250 may be a neural network, but any othersuitable types of machine-learning models may also be used.

In particular embodiments, the ranking system 200, in operation or atinference time, may use the trained machine-learning model 250 togenerate a second preliminary rank for a given content item 211 based onits custom attributes 213. In particular embodiments, attributes of thecontent item 211 that are of the custom types 213 may be processed by aclustering model 240. Then, those custom attributes and/or theirrespective cluster representations generated by the clustering model 240may be input into the trained machine-learning model 250. In particularembodiments where the machine-learning model 250 further considers userdata, the system 200 may retrieve 260 the user data (e.g., based on theuser ID 214) from the social-networking/content-distribution platformand/or the content provider. Based on these input data, themachine-learning model 250 may generate a ranking score for thecorresponding content item 211 (e.g., the ranking score may represent arelative relevance of the content item to the particular user). Theranking score from the machine-learning model 250 may then be input intoa final ranking model 270, which will be described below

In particular embodiments, the ranking system 200 may use a thirdmachine-learning ranking model 270 that is trained to output a rankingscore based on the outputs of the machine-learning model 230 for knownattributes and the machine-learning model 250 for custom attributes. Thethird ranking model 270 may be configured to take as input the outputsof model 230 and model 250. Conceptually, the ranking model 270 istrained to predict a ranking score based on (1) the ranking scoregenerated based on known attributes and (2) the ranking score generatedbased on custom attributes). In particular embodiments, the thirdranking model 270 may take into consideration information about contentproviders (e.g., characteristics such as size, industry type, etc.) andtheir ranking objectives (e.g., optimizing engagement or time spent,target demographics, etc.). Additionally or alternatively, the thirdranking model 270 may also take into account context information aboutusers 215 (e.g., recently viewed items, etc.). Thus, in particularembodiments, each training sample in the training dataset used fortraining the ranking model 270 may further include (1) contextual oractivity information 215 associated with the user to whom the trainingcontent item of the training sample was presented, and/or (2) metadatarelating to the content provider 216 of the training content item of thetraining sample. As with the other aforementioned machine-learningmodels (e.g., the machine-learning model 230 for known attributes, theclustering model 240, the machine-learning model 250 for clusterrepresentations), the ranking model 270 may be trained using dataassociated with content providers different from the content providerwho is at inference time requesting the system to rank its contentitems. At inference time, the ranking model 270 may, therefore, takeinto all the aforementioned considerations when making a determinationas to how a given content item 211 should be ranked given a particularuser 214. In particular embodiments, the ranking model 270 may output avalue (e.g., between 0 and 1) that represents how likely the user 214would behave in the desired manner in response to being presented withthe content item 211.

To further fine-tune ranking results, small models customized for eachcontent provider may be trained based on the available data (even iflimited). For example, a personalized machine-learning model may betrained for a particular content provider using both the known attributetypes 212 and the custom attribute types 213 of the content provider'scontent items 211. Since the amount of training data may be limited(which may be deliberately limited to minimize training time or simplydue to limited availability), the accuracy of the personalized model maybe insufficient. However, while the ranking outcome of suchcustom-tailored model may not have the benefit of being trained on therich, global data of the social-networking/content-distributionplatform, it may nevertheless provide some benefit in determiningranking. As such, in particular embodiments, the ranking model 270 mayfurther be trained to take into consideration the ranking outputs ofpersonalized models.

FIG. 3 illustrates an example method illustrates an example method 300for ranking a content item using the ranking system. The method maybegin at step 310, where a computing system associated with the rankingsystem may receive a request to rank content items for users. Thecomputing system may receive the request directly from a contentprovider of the content items (e.g., via an API call). The computingsystem may also indirectly receive the request from the content providervia other handlers of the system. For example, the content provider mayhave issued a general request to distribute content items to users whenthey log into the system. Thus, whenever a user meeting the desiredcharacteristics (e.g., demographics, interests, etc.), the system mayautomatically invoke the ranking system to determine which of thecontent items would be ranked highly with respect to that user.

At step 320, the computing system may access a content item associatedwith the requesting content provider. For example, the content item maybe stored in a database or dynamically received and stored at leasttemporarily in RAM. In particular embodiments, the content item may havea first set of attributes (e.g., of known attribute types) and a secondset of attributes (e.g., of custom attribute types). For example, if thecontent item is a song, the first set of attributes may include a songtitle/label and description, and the second set of attributes mayinclude song duration. In particular embodiments, data associated withthe content item may be structured (e.g., in JSON or XML format), andbased on the structured information, the system may identify theattributes with known attribute types. Attributes that are not or cannotbe mapped to the known attribute types may be deemed custom attributes.

At step 330, the computing system may generate, using a firstmachine-learning model that is trained/configured to process attributesof the known type, a first ranking score of the content item for a userbased on the first set of attributes. In particular embodiments, thesystem may map or transform the first set of attributes of the knowntype into a format that the first machine-learning model was trained toprocess. As an example, certain attributes may have a one-to-one mappingwith a known attribute type (e.g., a song title may have a one-to-onemapping with an item label). Other attributes of the content item may betransformed to conform to the known attribute type (e.g., a content itemhaving a separate “depth,” “height,” and “width” attributes may betransformed to form a single known “dimension” attribute, which may havethe format: height x width x depth). In particular embodiments, thefirst machine-learning model may process the known attributes of thecontent item, without considering data associated with any particularuser, and output a ranking score. The ranking score may represent thelikely relative desirability of the content item relative to othercontent items. In particular embodiments, the first machine-learningmodel may also take as input user data associated with a particular userfor whom the content item is being ranked. In such a case, the systemmay retrieve the user data from the content provider, thesocial-networking/content-distribution platform, or any other suitablesource where such data is available. After processing the content item'sknown attributes and the user data, the first machine-learning model mayoutput a ranking score. The ranking score may represent, for example, alikely suitability of the content item for that particular user. Theranking score generated by the first machine-learning model may beconsidered as preliminary since the first machine-learning model did notconsider the custom attributes of the content item.

At step 340, the system may generate cluster representations of thesecond set of attributes (custom attributes) of the content item. Asdescribed elsewhere herein, particular embodiments may use a clusteringmodel trained to generalize custom attributes so that attributes placedin the same cluster would be more similar to each other than to those inother clusters. In particular embodiments, the clustering model mayimplement any suitable clustering algorithms (e.g., k-means,hierarchical clustering, etc.) to perform the task of clustering. Theoutput of the clustering model may be cluster representations of theinput custom attributes.

At step 350, the system may generate, using a second machine-learningmodel, a second ranking score of the content item for the user based onthe cluster representations. In particular embodiments, the secondmachine-learning model may have been trained on cluster representationsof a variety of custom attributes of different types of training contentitems. As such, the second machine-learning model has learned how toprocess the cluster representations of the content item and infer theappropriate ranking score therefrom. In particular embodiments, thesecond machine-learning model may have further been trained to take intoconsideration a user's data when determining the appropriate rankingscore for that user. In this case, the second machine-learning model mayfurther take as input the user data of the user for whom the ranking isbeing generated. The ranking score output by the second machine-learningmodel may represent, for example, a likely suitability of the contentitem for that particular user. Similar to the ranking score generated bythe first machine-learning model, the ranking score from the secondmachine-learning model may be considered to be preliminary, since thesecond machine-learning model may not take into consideration the knownattributes of the content item.

At step 360, the system may generate, using a third machine-learningmodel, a third ranking score of the content item for the user based onthe first ranking score and the second ranking score from the firstmachine-learning model and the second machine-learning model,respectively. Conceptually, the third machine-learning model is onelayer of abstraction removed from the underlying attributes of thecontent item. Rather than processing the attributes of the content itemdirectly, it may do so indirectly through the outputs of the first andsecond machine-learning models. In particular embodiments the thirdmachine-learning model may also consider predetermined types of metadataassociated with the content provider of the content item since thecharacteristics of the content provider may be predictive of the user'slevel of interest in the content item. For example, a particular usermay only be interested in sports news from a news source that handlessports news exclusively and not from a more generic news source that, inaddition to sports, reports on financial news, foreign news, domesticnews, etc. Examples of the content provider's metadata may include thesize and industry of the content provider, target outcomes (e.g.,ranking objectives) for presenting the content item to the user, targetdemographics of the user, and/or any other suitable information. Inparticular embodiments, the third machine-learning model mayadditionally or alternatively take as input context informationassociated with the particular user for whom the ranking is beinggenerated. In particular embodiments, the context information mayprovide the third machine-learning model with a sense of the context inwhich the content item may be presented. For example, if the contentprovider wishes to determine which content item to present to aparticular user in the near future, context information that may reflectthe user's current or recent state of mind or interest may be predictiveof which content item would more likely to be of interest to the user.Thus, in particular embodiments, the third machine-learning model mayfurther take as input predetermined types of context informationassociated with the user in generating the ranking score. The rankingscore generated by the third machine-learning model may be considered asthe final ranking score for the content item.

At step 370, the system may determine whether the ranking scoregenerated by the third machine-learning model satisfies one or morepredetermined criteria. For example, the system may compare the rankingscore of the content item with the ranking scores of other contentitems, respectively, to determine which of the content items have thehighest-ranking scores. As another example, the system may determinewhether the ranking score generated by the third machine-learning modelis above a certain predetermined threshold value. If any of the contentitems satisfy the one or more criteria, the system may proceed, at step380, to select the content item for presentation to the user based onthe third ranking score. This may include, for example, surfacinginformation associated with the content item (e.g., a banner display,video, audio, or any other perceptible information relating to thecontent item) to the user through the social-networking platform orcontent distribution platform. The system may also push notifications(e.g., via e-mail, SMS text, in-app notifications, etc.) relating to thecontent item to the user's device. The system may also inform thecontent provider of the ranking results (e.g., the IDs of the contentitems that satisfied the threshold criteria) and let the contentprovider decide how to use the information. The content provider maysimilarly display banners and/or other forms of perceptible messages tothe user via its own platform.

Particular embodiments may repeat one or more steps of the method ofFIG. 3, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 3 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 3 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method for rankinga content item using the ranking system, including the particular stepsof the method of FIG. 3, this disclosure contemplates any suitablemethod for ranking a content item using the ranking system, includingany suitable steps, which may include all, some, or none of the steps ofthe method of FIG. 3, where appropriate. Furthermore, although thisdisclosure describes and illustrates particular components, devices, orsystems carrying out particular steps of the method of FIG. 3, thisdisclosure contemplates any suitable combination of any suitablecomponents, devices, or systems carrying out any suitable steps of themethod of FIG. 3.

FIG. 4 illustrates an example network environment 400 associated with asocial-networking system. Network environment 400 includes a clientsystem 430, a social-networking system 460, and a third-party system 470connected to each other by a network 410. Although FIG. 4 illustrates aparticular arrangement of client system 430, social-networking system460, third-party system 470, and network 410, this disclosurecontemplates any suitable arrangement of client system 430,social-networking system 460, third-party system 470, and network 410.As an example and not by way of limitation, two or more of client system430, social-networking system 460, and third-party system 470 may beconnected to each other directly, bypassing network 410. As anotherexample, two or more of client system 430, social-networking system 460,and third-party system 470 may be physically or logically co-locatedwith each other in whole or in part. Moreover, although FIG. 4illustrates a particular number of client systems 430, social-networkingsystems 460, third-party systems 470, and networks 410, this disclosurecontemplates any suitable number of client systems 430,social-networking systems 460, third-party systems 470, and networks410. As an example and not by way of limitation, network environment 400may include multiple client systems 430, social-networking systems 460,third-party systems 470, and networks 410.

This disclosure contemplates any suitable network 410. As an example andnot by way of limitation, one or more portions of network 410 mayinclude an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, or a combinationof two or more of these. Network 410 may include one or more networks410.

Links 450 may connect client system 430, social-networking system 460,and third-party system 470 to communication network 410 or to eachother. This disclosure contemplates any suitable links 450. Inparticular embodiments, one or more links 450 include one or morewireline (such as for example Digital Subscriber Line (DSL) or Data OverCable Service Interface Specification (DOC SIS)), wireless (such as forexample Wi-Fi or Worldwide Interoperability for Microwave Access(WiMAX)), or optical (such as for example Synchronous Optical Network(SONET) or Synchronous Digital Hierarchy (SDH)) links. In particularembodiments, one or more links 450 each include an ad hoc network, anintranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, aportion of the Internet, a portion of the PSTN, a cellulartechnology-based network, a satellite communications technology-basednetwork, another link 450, or a combination of two or more such links450. Links 450 need not necessarily be the same throughout networkenvironment 400. One or more first links 450 may differ in one or morerespects from one or more second links 450.

In particular embodiments, client system 430 may be an electronic deviceincluding hardware, software, or embedded logic components or acombination of two or more such components and capable of carrying outthe appropriate functionalities implemented or supported by clientsystem 430. As an example and not by way of limitation, a client system430 may include a computer system such as a desktop computer, notebookor laptop computer, netbook, a tablet computer, e-book reader, GPSdevice, camera, personal digital assistant (PDA), handheld electronicdevice, cellular telephone, smartphone, augmented/virtual realitydevice, other suitable electronic device, or any suitable combinationthereof. This disclosure contemplates any suitable client systems 430. Aclient system 430 may enable a network user at client system 430 toaccess network 410. A client system 430 may enable its user tocommunicate with other users at other client systems 430.

In particular embodiments, client system 430 may include a web browser432, such as MICROSOFT INTERNET EXPLORER, GOOGLE CHROME or MOZILLAFIREFOX, and may have one or more add-ons, plug-ins, or otherextensions, such as TOOLBAR or YAHOO TOOLBAR. A user at client system430 may enter a Uniform Resource Locator (URL) or other addressdirecting the web browser 432 to a particular server (such as server462, or a server associated with a third-party system 470), and the webbrowser 432 may generate a Hyper Text Transfer Protocol (HTTP) requestand communicate the HTTP request to server. The server may accept theHTTP request and communicate to client system 430 one or more Hyper TextMarkup Language (HTML) files responsive to the HTTP request. Clientsystem 430 may render a webpage based on the HTML files from the serverfor presentation to the user. This disclosure contemplates any suitablewebpage files. As an example and not by way of limitation, webpages mayrender from HTML files, Extensible Hyper Text Markup Language (XHTML)files, or Extensible Markup Language (XML) files, according toparticular needs. Such pages may also execute scripts such as, forexample and without limitation, those written in JAVASCRIPT, JAVA,MICROSOFT SILVERLIGHT, combinations of markup language and scripts suchas AJAX (Asynchronous JAVASCRIPT and XML), and the like. Herein,reference to a webpage encompasses one or more corresponding webpagefiles (which a browser may use to render the webpage) and vice versa,where appropriate.

In particular embodiments, social-networking system 460 may be anetwork-addressable computing system that can host an online socialnetwork. Social-networking system 460 may generate, store, receive, andsend social-networking data, such as, for example, user-profile data,concept-profile data, social-graph information, or other suitable datarelated to the online social network. Social-networking system 460 maybe accessed by the other components of network environment 400 eitherdirectly or via network 410. As an example and not by way of limitation,client system 430 may access social-networking system 460 using a webbrowser 432, or a native application associated with social-networkingsystem 460 (e.g., a mobile social-networking application, a messagingapplication, another suitable application, or any combination thereof)either directly or via network 410. In particular embodiments,social-networking system 460 may include one or more servers 462. Eachserver 462 may be a unitary server or a distributed server spanningmultiple computers or multiple datacenters. Servers 462 may be ofvarious types, such as, for example and without limitation, web server,news server, mail server, message server, advertising server, fileserver, application server, exchange server, database server, proxyserver, another server suitable for performing functions or processesdescribed herein, or any combination thereof. In particular embodiments,each server 462 may include hardware, software, or embedded logiccomponents or a combination of two or more such components for carryingout the appropriate functionalities implemented or supported by server462. In particular embodiments, social-networking system 460 may includeone or more data stores 464. Data stores 464 may be used to storevarious types of information. In particular embodiments, the informationstored in data stores 464 may be organized according to specific datastructures. In particular embodiments, each data store 464 may be arelational, columnar, correlation, or other suitable database. Althoughthis disclosure describes or illustrates particular types of databases,this disclosure contemplates any suitable types of databases. Particularembodiments may provide interfaces that enable a client system 430, asocial-networking system 460, or a third-party system 470 to manage,retrieve, modify, add, or delete, the information stored in data store464.

In particular embodiments, social-networking system 460 may store one ormore social graphs in one or more data stores 464. In particularembodiments, a social graph may include multiple nodes—which may includemultiple user nodes (each corresponding to a particular user) ormultiple concept nodes (each corresponding to a particular concept)—andmultiple edges connecting the nodes. Social-networking system 460 mayprovide users of the online social network the ability to communicateand interact with other users. In particular embodiments, users may jointhe online social network via social-networking system 460 and then addconnections (e.g., relationships) to a number of other users ofsocial-networking system 460 to whom they want to be connected. Herein,the term “friend” may refer to any other user of social-networkingsystem 460 with whom a user has formed a connection, association, orrelationship via social-networking system 460.

In particular embodiments, social-networking system 460 may provideusers with the ability to take actions on various types of items orobjects, supported by social-networking system 460. As an example andnot by way of limitation, the items and objects may include groups orsocial networks to which users of social-networking system 460 maybelong, events or calendar entries in which a user might be interested,computer-based applications that a user may use, transactions that allowusers to buy or sell items via the service, interactions withadvertisements that a user may perform, or other suitable items orobjects. A user may interact with anything that is capable of beingrepresented in social-networking system 460 or by an external system ofthird-party system 470, which is separate from social-networking system460 and coupled to social-networking system 460 via a network 410.

In particular embodiments, social-networking system 460 may be capableof linking a variety of entities. As an example and not by way oflimitation, social-networking system 460 may enable users to interactwith each other as well as receive content from third-party systems 470or other entities, or to allow users to interact with these entitiesthrough an application programming interfaces (API) or othercommunication channels.

In particular embodiments, a third-party system 470 may include one ormore types of servers, one or more data stores, one or more interfaces,including but not limited to APIs, one or more web services, one or morecontent sources, one or more networks, or any other suitable components,e.g., that servers may communicate with. A third-party system 470 may beoperated by a different entity from an entity operatingsocial-networking system 460. In particular embodiments, however,social-networking system 460 and third-party systems 470 may operate inconjunction with each other to provide social-networking services tousers of social-networking system 460 or third-party systems 470. Inthis sense, social-networking system 460 may provide a platform, orbackbone, which other systems, such as third-party systems 470, may useto provide social-networking services and functionality to users acrossthe Internet.

In particular embodiments, a third-party system 470 may include athird-party content object provider. A third-party content objectprovider may include one or more sources of content objects, which maybe communicated to a client system 430. As an example and not by way oflimitation, content objects may include information regarding things oractivities of interest to the user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, productinformation and reviews, or other suitable information. As anotherexample and not by way of limitation, content objects may includeincentive content objects, such as coupons, discount tickets, giftcertificates, or other suitable incentive objects.

In particular embodiments, social-networking system 460 also includesuser-generated content objects, which may enhance a user's interactionswith social-networking system 460. User-generated content may includeanything a user can add, upload, send, or “post” to social-networkingsystem 460. As an example and not by way of limitation, a usercommunicates posts to social-networking system 460 from a client system430. Posts may include data such as status updates or other textualdata, location information, photos, videos, links, music or othersimilar data or media. Content may also be added to social-networkingsystem 460 by a third-party through a “communication channel,” such as anewsfeed or stream.

In particular embodiments, social-networking system 460 may include avariety of servers, sub-systems, programs, modules, logs, and datastores. In particular embodiments, social-networking system 460 mayinclude one or more of the following: a web server, action logger,API-request server, relevance-and-ranking engine, content-objectclassifier, notification controller, action log,third-party-content-object-exposure log, inference module,authorization/privacy server, search module, advertisement-targetingmodule, user-interface module, user-profile store, connection store,third-party content store, or location store. Social-networking system460 may also include suitable components such as network interfaces,security mechanisms, load balancers, failover servers,management-and-network-operations consoles, other suitable components,or any suitable combination thereof. In particular embodiments,social-networking system 460 may include one or more user-profile storesfor storing user profiles. A user profile may include, for example,biographic information, demographic information, behavioral information,social information, or other types of descriptive information, such aswork experience, educational history, hobbies or preferences, interests,affinities, or location. Interest information may include interestsrelated to one or more categories. Categories may be general orspecific. As an example and not by way of limitation, if a user “likes”an article about a brand of shoes the category may be the brand, or thegeneral category of “shoes” or “clothing.” A connection store may beused for storing connection information about users. The connectioninformation may indicate users who have similar or common workexperience, group memberships, hobbies, educational history, or are inany way related or share common attributes. The connection informationmay also include user-defined connections between different users andcontent (both internal and external). A web server may be used forlinking social-networking system 460 to one or more client systems 430or one or more third-party system 470 via network 410. The web servermay include a mail server or other messaging functionality for receivingand routing messages between social-networking system 460 and one ormore client systems 430. An API-request server may allow a third-partysystem 470 to access information from social-networking system 460 bycalling one or more APIs. An action logger may be used to receivecommunications from a web server about a user's actions on or offsocial-networking system 460. In conjunction with the action log, athird-party-content-object log may be maintained of user exposures tothird-party-content objects. A notification controller may provideinformation regarding content objects to a client system 430.Information may be pushed to a client system 430 as notifications, orinformation may be pulled from client system 430 responsive to a requestreceived from client system 430. Authorization servers may be used toenforce one or more privacy settings of the users of social-networkingsystem 460. A privacy setting of a user determines how particularinformation associated with a user can be shared. The authorizationserver may allow users to opt in to or opt out of having their actionslogged by social-networking system 460 or shared with other systems(e.g., third-party system 470), such as, for example, by settingappropriate privacy settings. Third-party-content-object stores may beused to store content objects received from third parties, such as athird-party system 470. Location stores may be used for storing locationinformation received from client systems 430 associated with users.Advertisement-pricing modules may combine social information, thecurrent time, location information, or other suitable information toprovide relevant advertisements, in the form of notifications, to auser.

FIG. 5 illustrates example social graph 500. In particular embodiments,social-networking system 460 may store one or more social graphs 500 inone or more data stores. In particular embodiments, social graph 500 mayinclude multiple nodes—which may include multiple user nodes 502 ormultiple concept nodes 504—and multiple edges 506 connecting the nodes.Example social graph 500 illustrated in FIG. 5 is shown, for didacticpurposes, in a two-dimensional visual map representation. In particularembodiments, a social-networking system 460, client system 430, orthird-party system 470 may access social graph 500 and relatedsocial-graph information for suitable applications. The nodes and edgesof social graph 500 may be stored as data objects, for example, in adata store (such as a social-graph database). Such a data store mayinclude one or more searchable or queryable indexes of nodes or edges ofsocial graph 500.

In particular embodiments, a user node 502 may correspond to a user ofsocial-networking system 460. As an example and not by way oflimitation, a user may be an individual (human user), an entity (e.g.,an enterprise, business, or third-party application), or a group (e.g.,of individuals or entities) that interacts or communicates with or oversocial-networking system 460. In particular embodiments, when a userregisters for an account with social-networking system 460,social-networking system 460 may create a user node 502 corresponding tothe user, and store the user node 502 in one or more data stores. Usersand user nodes 502 described herein may, where appropriate, refer toregistered users and user nodes 502 associated with registered users. Inaddition or as an alternative, users and user nodes 502 described hereinmay, where appropriate, refer to users that have not registered withsocial-networking system 460. In particular embodiments, a user node 502may be associated with information provided by a user or informationgathered by various systems, including social-networking system 460. Asan example and not by way of limitation, a user may provide his or hername, profile picture, contact information, birth date, sex, maritalstatus, family status, employment, education background, preferences,interests, or other demographic information. In particular embodiments,a user node 502 may be associated with one or more data objectscorresponding to information associated with a user. In particularembodiments, a user node 502 may correspond to one or more webpages.

In particular embodiments, a concept node 504 may correspond to aconcept. As an example and not by way of limitation, a concept maycorrespond to a place (such as, for example, a movie theater,restaurant, landmark, or city); a website (such as, for example, awebsite associated with social-network system 460 or a third-partywebsite associated with a web-application server); an entity (such as,for example, a person, business, group, sports team, or celebrity); aresource (such as, for example, an audio file, video file, digitalphoto, text file, structured document, or application) which may belocated within social-networking system 460 or on an external server,such as a web-application server; real or intellectual property (suchas, for example, a sculpture, painting, movie, game, song, idea,photograph, or written work); a game; an activity; an idea or theory; anobject in a augmented/virtual reality environment; another suitableconcept; or two or more such concepts. A concept node 504 may beassociated with information of a concept provided by a user orinformation gathered by various systems, including social-networkingsystem 460. As an example and not by way of limitation, information of aconcept may include a name or a title; one or more images (e.g., animage of the cover page of a book); a location (e.g., an address or ageographical location); a website (which may be associated with a URL);contact information (e.g., a phone number or an email address); othersuitable concept information; or any suitable combination of suchinformation. In particular embodiments, a concept node 504 may beassociated with one or more data objects corresponding to informationassociated with concept node 504. In particular embodiments, a conceptnode 504 may correspond to one or more webpages.

In particular embodiments, a node in social graph 500 may represent orbe represented by a webpage (which may be referred to as a “profilepage”). Profile pages may be hosted by or accessible tosocial-networking system 460. Profile pages may also be hosted onthird-party websites associated with a third-party system 470. As anexample and not by way of limitation, a profile page corresponding to aparticular external webpage may be the particular external webpage andthe profile page may correspond to a particular concept node 504.Profile pages may be viewable by all or a selected subset of otherusers. As an example and not by way of limitation, a user node 502 mayhave a corresponding user-profile page in which the corresponding usermay add content, make declarations, or otherwise express himself orherself. As another example and not by way of limitation, a concept node504 may have a corresponding concept-profile page in which one or moreusers may add content, make declarations, or express themselves,particularly in relation to the concept corresponding to concept node504.

In particular embodiments, a concept node 504 may represent athird-party webpage or resource hosted by a third-party system 470. Thethird-party webpage or resource may include, among other elements,content, a selectable or other icon, or other inter-actable object(which may be implemented, for example, in JavaScript, AJAX, or PHPcodes) representing an action or activity. As an example and not by wayof limitation, a third-party webpage may include a selectable icon suchas “like,” “check-in,” “eat,” “recommend,” or another suitable action oractivity. A user viewing the third-party webpage may perform an actionby selecting one of the icons (e.g., “check-in”), causing a clientsystem 430 to send to social-networking system 460 a message indicatingthe user's action. In response to the message, social-networking system460 may create an edge (e.g., a check-in-type edge) between a user node502 corresponding to the user and a concept node 504 corresponding tothe third-party webpage or resource and store edge 506 in one or moredata stores.

In particular embodiments, a pair of nodes in social graph 500 may beconnected to each other by one or more edges 506. An edge 506 connectinga pair of nodes may represent a relationship between the pair of nodes.In particular embodiments, an edge 506 may include or represent one ormore data objects or attributes corresponding to the relationshipbetween a pair of nodes. As an example and not by way of limitation, afirst user may indicate that a second user is a “friend” of the firstuser. In response to this indication, social-networking system 460 maysend a “friend request” to the second user. If the second user confirmsthe “friend request,” social-networking system 460 may create an edge506 connecting the first user's user node 502 to the second user's usernode 502 in social graph 500 and store edge 506 as social-graphinformation in one or more of data stores 464. In the example of FIG. 5,social graph 500 includes an edge 506 indicating a friend relationbetween user nodes 502 of user “A” and user “B” and an edge indicating afriend relation between user nodes 502 of user “C” and user “B.”Although this disclosure describes or illustrates particular edges 506with particular attributes connecting particular user nodes 502, thisdisclosure contemplates any suitable edges 506 with any suitableattributes connecting user nodes 502. As an example and not by way oflimitation, an edge 506 may represent a friendship, family relationship,business or employment relationship, fan relationship (including, e.g.,liking, etc.), follower relationship, visitor relationship (including,e.g., accessing, viewing, checking-in, sharing, etc.), subscriberrelationship, superior/subordinate relationship, reciprocalrelationship, non-reciprocal relationship, another suitable type ofrelationship, or two or more such relationships. Moreover, although thisdisclosure generally describes nodes as being connected, this disclosurealso describes users or concepts as being connected. Herein, referencesto users or concepts being connected may, where appropriate, refer tothe nodes corresponding to those users or concepts being connected insocial graph 500 by one or more edges 506. The degree of separationbetween two objects represented by two nodes, respectively, is a countof edges in a shortest path connecting the two nodes in the social graph500. As an example and not by way of limitation, in the social graph500, the user node 502 of user “C” is connected to the user node 502 ofuser “A” via multiple paths including, for example, a first pathdirectly passing through the user node 502 of user “B,” a second pathpassing through the concept node 504 of company “Acme” and the user node502 of user “D,” and a third path passing through the user nodes 502 andconcept nodes 504 representing school “Stanford,” user “G,” company“Acme,” and user “D.” User “C” and user “A” have a degree of separationof two because the shortest path connecting their corresponding nodes(i.e., the first path) includes two edges 506.

In particular embodiments, an edge 506 between a user node 502 and aconcept node 504 may represent a particular action or activity performedby a user associated with user node 502 toward a concept associated witha concept node 504. As an example and not by way of limitation, asillustrated in FIG. 5, a user may “like,” “attended,” “played,”“listened,” “cooked,” “worked at,” or “watched” a concept, each of whichmay correspond to an edge type or subtype. A concept-profile pagecorresponding to a concept node 504 may include, for example, aselectable “check in” icon (such as, for example, a clickable “check in”icon) or a selectable “add to favorites” icon. Similarly, after a userclicks these icons, social-networking system 460 may create a “favorite”edge or a “check in” edge in response to a user's action correspondingto a respective action. As another example and not by way of limitation,a user (user “C”) may listen to a particular song (“Imagine”) using aparticular application (SPOTIFY, which is an online music application).In this case, social-networking system 460 may create a “listened” edge506 and a “used” edge (as illustrated in FIG. 5) between user nodes 502corresponding to the user and concept nodes 504 corresponding to thesong and application to indicate that the user listened to the song andused the application. Moreover, social-networking system 460 may createa “played” edge 506 (as illustrated in FIG. 5) between concept nodes 504corresponding to the song and the application to indicate that theparticular song was played by the particular application. In this case,“played” edge 506 corresponds to an action performed by an externalapplication (SPOTIFY) on an external audio file (the song “Imagine”).Although this disclosure describes particular edges 506 with particularattributes connecting user nodes 502 and concept nodes 504, thisdisclosure contemplates any suitable edges 506 with any suitableattributes connecting user nodes 502 and concept nodes 504. Moreover,although this disclosure describes edges between a user node 502 and aconcept node 504 representing a single relationship, this disclosurecontemplates edges between a user node 502 and a concept node 504representing one or more relationships. As an example and not by way oflimitation, an edge 506 may represent both that a user likes and hasused at a particular concept. Alternatively, another edge 506 mayrepresent each type of relationship (or multiples of a singlerelationship) between a user node 502 and a concept node 504 (asillustrated in FIG. 5 between user node 502 for user “E” and conceptnode 504 for “SPOTIFY”).

In particular embodiments, social-networking system 460 may create anedge 506 between a user node 502 and a concept node 504 in social graph500. As an example and not by way of limitation, a user viewing aconcept-profile page (such as, for example, by using a web browser or aspecial-purpose application hosted by the user's client system 430) mayindicate that he or she likes the concept represented by the conceptnode 504 by clicking or selecting a “Like” icon, which may cause theuser's client system 430 to send to social-networking system 460 amessage indicating the user's liking of the concept associated with theconcept-profile page. In response to the message, social-networkingsystem 460 may create an edge 506 between user node 502 associated withthe user and concept node 504, as illustrated by “like” edge 506 betweenthe user and concept node 504. In particular embodiments,social-networking system 460 may store an edge 506 in one or more datastores. In particular embodiments, an edge 506 may be automaticallyformed by social-networking system 460 in response to a particular useraction. As an example and not by way of limitation, if a first useruploads a picture, watches a movie, or listens to a song, an edge 506may be formed between user node 502 corresponding to the first user andconcept nodes 504 corresponding to those concepts. Although thisdisclosure describes forming particular edges 506 in particular manners,this disclosure contemplates forming any suitable edges 506 in anysuitable manner.

In particular embodiments, social-networking system 460 may determinethe social-graph affinity (which may be referred to herein as“affinity”) of various social-graph entities for each other. Affinitymay represent the strength of a relationship or level of interestbetween particular objects associated with the online social network,such as users, concepts, content, actions, advertisements, other objectsassociated with the online social network, or any suitable combinationthereof. Affinity may also be determined with respect to objectsassociated with third-party systems 470 or other suitable systems. Anoverall affinity for a social-graph entity for each user, subjectmatter, or type of content may be established. The overall affinity maychange based on continued monitoring of the actions or relationshipsassociated with the social-graph entity. Although this disclosuredescribes determining particular affinities in a particular manner, thisdisclosure contemplates determining any suitable affinities in anysuitable manner.

In particular embodiments, social-networking system 460 may measure orquantify social-graph affinity using an affinity coefficient (which maybe referred to herein as “coefficient”). The coefficient may representor quantify the strength of a relationship between particular objectsassociated with the online social network. The coefficient may alsorepresent a probability or function that measures a predictedprobability that a user will perform a particular action based on theuser's interest in the action. In this way, a user's future actions maybe predicted based on the user's prior actions, where the coefficientmay be calculated at least in part on the history of the user's actions.Coefficients may be used to predict any number of actions, which may bewithin or outside of the online social network. As an example and not byway of limitation, these actions may include various types ofcommunications, such as sending messages, posting content, or commentingon content; various types of observation actions, such as accessing orviewing profile pages, media, or other suitable content; various typesof coincidence information about two or more social-graph entities, suchas being in the same group, tagged in the same photograph, checked-in atthe same location, or attending the same event; or other suitableactions. Although this disclosure describes measuring affinity in aparticular manner, this disclosure contemplates measuring affinity inany suitable manner.

In particular embodiments, social-networking system 460 may use avariety of factors to calculate a coefficient. These factors mayinclude, for example, user actions, types of relationships betweenobjects, location information, other suitable factors, or anycombination thereof. In particular embodiments, different factors may beweighted differently when calculating the coefficient. The weights foreach factor may be static or the weights may change according to, forexample, the user, the type of relationship, the type of action, theuser's location, and so forth. Ratings for the factors may be combinedaccording to their weights to determine an overall coefficient for theuser. As an example and not by way of limitation, particular useractions may be assigned both a rating and a weight while a relationshipassociated with the particular user action is assigned a rating and acorrelating weight (e.g., so the weights total 100%). To calculate thecoefficient of a user towards a particular object, the rating assignedto the user's actions may comprise, for example, 60% of the overallcoefficient, while the relationship between the user and the object maycomprise 40% of the overall coefficient. In particular embodiments, thesocial-networking system 460 may consider a variety of variables whendetermining weights for various factors used to calculate a coefficient,such as, for example, the time since information was accessed, decayfactors, frequency of access, relationship to information orrelationship to the object about which information was accessed,relationship to social-graph entities connected to the object, short- orlong-term averages of user actions, user feedback, other suitablevariables, or any combination thereof. As an example and not by way oflimitation, a coefficient may include a decay factor that causes thestrength of the signal provided by particular actions to decay withtime, such that more recent actions are more relevant when calculatingthe coefficient. The ratings and weights may be continuously updatedbased on continued tracking of the actions upon which the coefficient isbased. Any type of process or algorithm may be employed for assigning,combining, averaging, and so forth the ratings for each factor and theweights assigned to the factors. In particular embodiments,social-networking system 460 may determine coefficients usingmachine-learning algorithms trained on historical actions and past userresponses, or data farmed from users by exposing them to various optionsand measuring responses. Although this disclosure describes calculatingcoefficients in a particular manner, this disclosure contemplatescalculating coefficients in any suitable manner.

In particular embodiments, social-networking system 460 may calculate acoefficient based on a user's actions. Social-networking system 460 maymonitor such actions on the online social network, on a third-partysystem 470, on other suitable systems, or any combination thereof. Anysuitable type of user actions may be tracked or monitored. Typical useractions include viewing profile pages, creating or posting content,interacting with content, tagging or being tagged in images, joininggroups, listing and confirming attendance at events, checking-in atlocations, liking particular pages, creating pages, and performing othertasks that facilitate social action. In particular embodiments,social-networking system 460 may calculate a coefficient based on theuser's actions with particular types of content. The content may beassociated with the online social network, a third-party system 470, oranother suitable system. The content may include users, profile pages,posts, news stories, headlines, instant messages, chat roomconversations, emails, advertisements, pictures, video, music, othersuitable objects, or any combination thereof. Social-networking system460 may analyze a user's actions to determine whether one or more of theactions indicate an affinity for subject matter, content, other users,and so forth. As an example and not by way of limitation, if a userfrequently posts content related to “coffee” or variants thereof,social-networking system 460 may determine the user has a highcoefficient with respect to the concept “coffee”. Particular actions ortypes of actions may be assigned a higher weight and/or rating thanother actions, which may affect the overall calculated coefficient. Asan example and not by way of limitation, if a first user emails a seconduser, the weight or the rating for the action may be higher than if thefirst user simply views the user-profile page for the second user.

In particular embodiments, social-networking system 460 may calculate acoefficient based on the type of relationship between particularobjects. Referencing the social graph 500, social-networking system 460may analyze the number and/or type of edges 506 connecting particularuser nodes 502 and concept nodes 504 when calculating a coefficient. Asan example and not by way of limitation, user nodes 502 that areconnected by a spouse-type edge (representing that the two users aremarried) may be assigned a higher coefficient than a user nodes 502 thatare connected by a friend-type edge. In other words, depending upon theweights assigned to the actions and relationships for the particularuser, the overall affinity may be determined to be higher for contentabout the user's spouse than for content about the user's friend. Inparticular embodiments, the relationships a user has with another objectmay affect the weights and/or the ratings of the user's actions withrespect to calculating the coefficient for that object. As an exampleand not by way of limitation, if a user is tagged in a first photo, butmerely likes a second photo, social-networking system 460 may determinethat the user has a higher coefficient with respect to the first photothan the second photo because having a tagged-in-type relationship withcontent may be assigned a higher weight and/or rating than having alike-type relationship with content. In particular embodiments,social-networking system 460 may calculate a coefficient for a firstuser based on the relationship one or more second users have with aparticular object. In other words, the connections and coefficientsother users have with an object may affect the first user's coefficientfor the object. As an example and not by way of limitation, if a firstuser is connected to or has a high coefficient for one or more secondusers, and those second users are connected to or have a highcoefficient for a particular object, social-networking system 460 maydetermine that the first user should also have a relatively highcoefficient for the particular object. In particular embodiments, thecoefficient may be based on the degree of separation between particularobjects. The lower coefficient may represent the decreasing likelihoodthat the first user will share an interest in content objects of theuser that is indirectly connected to the first user in the social graph500. As an example and not by way of limitation, social-graph entitiesthat are closer in the social graph 500 (i.e., fewer degrees ofseparation) may have a higher coefficient than entities that are furtherapart in the social graph 500.

In particular embodiments, social-networking system 460 may calculate acoefficient based on location information. Objects that aregeographically closer to each other may be considered to be more relatedor of more interest to each other than more distant objects. Inparticular embodiments, the coefficient of a user towards a particularobject may be based on the proximity of the object's location to acurrent location associated with the user (or the location of a clientsystem 430 of the user). A first user may be more interested in otherusers or concepts that are closer to the first user. As an example andnot by way of limitation, if a user is one mile from an airport and twomiles from a gas station, social-networking system 460 may determinethat the user has a higher coefficient for the airport than the gasstation based on the proximity of the airport to the user.

In particular embodiments, social-networking system 460 may performparticular actions with respect to a user based on coefficientinformation. Coefficients may be used to predict whether a user willperform a particular action based on the user's interest in the action.A coefficient may be used when generating or presenting any type ofobjects to a user, such as advertisements, search results, news stories,media, messages, notifications, or other suitable objects. Thecoefficient may also be utilized to rank and order such objects, asappropriate. In this way, social-networking system 460 may provideinformation that is relevant to user's interests and currentcircumstances, increasing the likelihood that they will find suchinformation of interest. In particular embodiments, social-networkingsystem 460 may generate content based on coefficient information.Content objects may be provided or selected based on coefficientsspecific to a user. As an example and not by way of limitation, thecoefficient may be used to generate media for the user, where the usermay be presented with media for which the user has a high overallcoefficient with respect to the media object. As another example and notby way of limitation, the coefficient may be used to generateadvertisements for the user, where the user may be presented withadvertisements for which the user has a high overall coefficient withrespect to the advertised object. In particular embodiments,social-networking system 460 may generate search results based oncoefficient information. Search results for a particular user may bescored or ranked based on the coefficient associated with the searchresults with respect to the querying user. As an example and not by wayof limitation, search results corresponding to objects with highercoefficients may be ranked higher on a search-results page than resultscorresponding to objects having lower coefficients.

In particular embodiments, social-networking system 460 may calculate acoefficient in response to a request for a coefficient from a particularsystem or process. To predict the likely actions a user may take (or maybe the subject of) in a given situation, any process may request acalculated coefficient for a user. The request may also include a set ofweights to use for various factors used to calculate the coefficient.This request may come from a process running on the online socialnetwork, from a third-party system 470 (e.g., via an API or othercommunication channel), or from another suitable system. In response tothe request, social-networking system 460 may calculate the coefficient(or access the coefficient information if it has previously beencalculated and stored). In particular embodiments, social-networkingsystem 460 may measure an affinity with respect to a particular process.Different processes (both internal and external to the online socialnetwork) may request a coefficient for a particular object or set ofobjects. Social-networking system 460 may provide a measure of affinitythat is relevant to the particular process that requested the measure ofaffinity. In this way, each process receives a measure of affinity thatis tailored for the different context in which the process will use themeasure of affinity.

In connection with social-graph affinity and affinity coefficients,particular embodiments may utilize one or more systems, components,elements, functions, methods, operations, or steps disclosed in U.S.patent application Ser. No. 11/503093, filed 11 Aug. 2006, U.S. patentapplication Ser. No. 12/977027, filed 22 Dec. 2010, U.S. patentapplication Ser. No. 12/978265, filed 23 Dec. 2010, and U.S. patentapplication Ser. No. 13/632869, filed 1 Oct. 2012, each of which isincorporated by reference.

FIG. 6 illustrates an example computer system 600. In particularembodiments, one or more computer systems 600 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 600 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 600 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 600.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems600. This disclosure contemplates computer system 600 taking anysuitable physical form. As example and not by way of limitation,computer system 600 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (such as, for example, acomputer-on-module (COM) or system-on-module (SOM)), a desktop computersystem, a laptop or notebook computer system, an interactive kiosk, amainframe, a mesh of computer systems, a mobile telephone, a personaldigital assistant (PDA), a server, a tablet computer system, anaugmented/virtual reality device, or a combination of two or more ofthese. Where appropriate, computer system 600 may include one or morecomputer systems 600; be unitary or distributed; span multiplelocations; span multiple machines; span multiple data centers; or residein a cloud, which may include one or more cloud components in one ormore networks. Where appropriate, one or more computer systems 600 mayperform without substantial spatial or temporal limitation one or moresteps of one or more methods described or illustrated herein. As anexample and not by way of limitation, one or more computer systems 600may perform in real time or in batch mode one or more steps of one ormore methods described or illustrated herein. One or more computersystems 600 may perform at different times or at different locations oneor more steps of one or more methods described or illustrated herein,where appropriate.

In particular embodiments, computer system 600 includes a processor 602,memory 604, storage 606, an input/output (I/O) interface 608, acommunication interface 610, and a bus 612. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 602 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 602 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 604, or storage 606; decode and execute them; andthen write one or more results to an internal register, an internalcache, memory 604, or storage 606. In particular embodiments, processor602 may include one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 602 including anysuitable number of any suitable internal caches, where appropriate. Asan example and not by way of limitation, processor 602 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 604 or storage 606, andthe instruction caches may speed up retrieval of those instructions byprocessor 602. Data in the data caches may be copies of data in memory604 or storage 606 for instructions executing at processor 602 tooperate on; the results of previous instructions executed at processor602 for access by subsequent instructions executing at processor 602 orfor writing to memory 604 or storage 606; or other suitable data. Thedata caches may speed up read or write operations by processor 602. TheTLBs may speed up virtual-address translation for processor 602. Inparticular embodiments, processor 602 may include one or more internalregisters for data, instructions, or addresses. This disclosurecontemplates processor 602 including any suitable number of any suitableinternal registers, where appropriate. Where appropriate, processor 602may include one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 602. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 604 includes main memory for storinginstructions for processor 602 to execute or data for processor 602 tooperate on. As an example and not by way of limitation, computer system600 may load instructions from storage 606 or another source (such as,for example, another computer system 600) to memory 604. Processor 602may then load the instructions from memory 604 to an internal registeror internal cache. To execute the instructions, processor 602 mayretrieve the instructions from the internal register or internal cacheand decode them. During or after execution of the instructions,processor 602 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor602 may then write one or more of those results to memory 604. Inparticular embodiments, processor 602 executes only instructions in oneor more internal registers or internal caches or in memory 604 (asopposed to storage 606 or elsewhere) and operates only on data in one ormore internal registers or internal caches or in memory 604 (as opposedto storage 606 or elsewhere). One or more memory buses (which may eachinclude an address bus and a data bus) may couple processor 602 tomemory 604. Bus 612 may include one or more memory buses, as describedbelow. In particular embodiments, one or more memory management units(MMUs) reside between processor 602 and memory 604 and facilitateaccesses to memory 604 requested by processor 602. In particularembodiments, memory 604 includes random access memory (RAM). This RAMmay be volatile memory, where appropriate. Where appropriate, this RAMmay be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 604 may include one ormore memories 604, where appropriate. Although this disclosure describesand illustrates particular memory, this disclosure contemplates anysuitable memory.

In particular embodiments, storage 606 includes mass storage for data orinstructions. As an example and not by way of limitation, storage 606may include a hard disk drive (HDD), a floppy disk drive, flash memory,an optical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Storage606 may include removable or non-removable (or fixed) media, whereappropriate. Storage 606 may be internal or external to computer system600, where appropriate. In particular embodiments, storage 606 isnon-volatile, solid-state memory. In particular embodiments, storage 606includes read-only memory (ROM). Where appropriate, this ROM may bemask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM),electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM),or flash memory or a combination of two or more of these. Thisdisclosure contemplates mass storage 606 taking any suitable physicalform. Storage 606 may include one or more storage control unitsfacilitating communication between processor 602 and storage 606, whereappropriate. Where appropriate, storage 606 may include one or morestorages 606. Although this disclosure describes and illustratesparticular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface 608 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 600 and one or more I/O devices. Computer system600 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 600. As an example and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 608 for them. Where appropriate, I/O interface 608 mayinclude one or more device or software drivers enabling processor 602 todrive one or more of these I/O devices. I/O interface 608 may includeone or more I/O interfaces 608, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 610 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 600 and one or more other computer systems 600 or one ormore networks. As an example and not by way of limitation, communicationinterface 610 may include a network interface controller (NIC) ornetwork adapter for communicating with an Ethernet or other wire-basednetwork or a wireless NIC (WNIC) or wireless adapter for communicatingwith a wireless network, such as a WI-FI network. This disclosurecontemplates any suitable network and any suitable communicationinterface 610 for it. As an example and not by way of limitation,computer system 600 may communicate with an ad hoc network, a personalarea network (PAN), a local area network (LAN), a wide area network(WAN), a metropolitan area network (MAN), or one or more portions of theInternet or a combination of two or more of these. One or more portionsof one or more of these networks may be wired or wireless. As anexample, computer system 600 may communicate with a wireless PAN (WPAN)(such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAXnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network), or other suitablewireless network or a combination of two or more of these. Computersystem 600 may include any suitable communication interface 610 for anyof these networks, where appropriate. Communication interface 610 mayinclude one or more communication interfaces 610, where appropriate.Although this disclosure describes and illustrates a particularcommunication interface, this disclosure contemplates any suitablecommunication interface.

In particular embodiments, bus 612 includes hardware, software, or bothcoupling components of computer system 600 to each other. As an exampleand not by way of limitation, bus 612 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 612may include one or more buses 612, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

What is claimed is:
 1. A method comprising, by a computing system: accessing a content item associated with a content provider, the content item having a first set of attributes and a second set of attributes; generating, using a first machine-learning model, a first ranking score of the content item for a user based on the first set of attributes; generating cluster representations of the second set of attributes of the content item; generating, using a second machine-learning model, a second ranking score of the content item for the user based on the cluster representations; generating, using a third machine-learning model, a third ranking score of the content item for the user based on the first ranking score and the second ranking score; and selecting the content item for presentation to the user based on the third ranking score.
 2. The method of claim 1, wherein the generating of the third ranking score is further based on metadata associated with the content provider.
 3. The method of claim 2, wherein the metadata associated with the content provider comprises at least one of: a size of the content provider, a target outcome for presenting the content item to the user, or a target demographic of the user.
 4. The method of claim 1, further comprising: receiving, from the content provider associated with the content item, a request to rank the content item for the user; and receiving, from the content provider, context information associated the user; wherein the generating of the third ranking score is further based on the context information associated with the user.
 5. The method of claim 1, further comprising: accessing user data associated with the user; wherein the generating of the first ranking score or the generating of the second ranking score is further based on the user data associated with the user.
 6. The method of claim 1, wherein the first machine-learning model, the second machine-learning model, or the third-machine learning model is trained using data associated with at least a second content provider, the second content provider being different from the content provider.
 7. The method of claim 1, wherein the first set of attributes have one or more known attribute types; wherein the first machine-learning model is trained based on training data having the one or more known attribute types.
 8. The method of claim 7, wherein the second set of attributes have one or more custom attribute types; wherein the one or more custom attributes types are different from the known attribute types of the training data used for training the first machine-learning model.
 9. The method of claim 1, further comprising: generating, using a fourth machine-learning model, a fourth ranking score of the content item for the user based on the first set of attributes and the second set of attributes of the content item; wherein the generating of the third ranking score of the content items is further based on the fourth ranking score.
 10. The method of claim 9, wherein the fourth machine-learning model is trained using training data associated with the content provider.
 11. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising: accessing a content item associated with a content provider, the content item having a first set of attributes and a second set of attributes; generating, using a first machine-learning model, a first ranking score of the content item for a user based on the first set of attributes; generating cluster representations of the second set of attributes of the content item; generating, using a second machine-learning model, a second ranking score of the content item for the user based on the cluster representations; generating, using a third machine-learning model, a third ranking score of the content item for the user based on the first ranking score and the second ranking score; and selecting the content item for presentation to the user based on the third ranking score.
 12. The system of claim 11, wherein the generating of the third ranking score is further based on metadata associated with the content provider.
 13. The system of claim 12, wherein the metadata associated with the content provider comprises at least one of: a size of the content provider, a target outcome for presenting the content item to the user, or a target demographic of the user.
 14. The system of claim 11, wherein the processors are further operable when executing the instructions to perform operations comprising: receiving, from the content provider associated with the content item, a request to rank the content item for the user; and receiving, from the content provider, context information associated the user; wherein the generating of the third ranking score is further based on the context information associated with the user.
 15. The system of claim 11, wherein the processors are further operable when executing the instructions to perform operations comprising: accessing user data associated with the user; wherein the generating of the first ranking score or the generating of the second ranking score is further based on the user data associated with the user.
 16. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising: accessing a content item associated with a content provider, the content item having a first set of attributes and a second set of attributes; generating, using a first machine-learning model, a first ranking score of the content item for a user based on the first set of attributes; generating cluster representations of the second set of attributes of the content item; generating, using a second machine-learning model, a second ranking score of the content item for the user based on the cluster representations; generating, using a third machine-learning model, a third ranking score of the content item for the user based on the first ranking score and the second ranking score; and selecting the content item for presentation to the user based on the third ranking score.
 17. The media of claim 16, wherein the generating of the third ranking score is further based on metadata associated with the content provider.
 18. The media of claim 17, wherein the metadata associated with the content provider comprises at least one of: a size of the content provider, a target outcome for presenting the content item to the user, or a target demographic of the user.
 19. The media of claim 16, wherein the software is further operable when executed to cause the one or more processors to perform operations comprising: receiving, from the content provider associated with the content item, a request to rank the content item for the user; and receiving, from the content provider, context information associated the user; wherein the generating of the third ranking score is further based on the context information associated with the user.
 20. The media of claim 16, wherein the software is further operable when executed to cause the one or more processors to perform operations comprising: accessing user data associated with the user; wherein the generating of the first ranking score or the generating of the second ranking score is further based on the user data associated with the user. 